Dishwasher and method for acquiring three-dimensional image thereof

ABSTRACT

A dishwasher includes: a tub, a sump disposed in the tub, a rack disposed in the sump and receiving an object, a plurality of lighting devices disposed at the tub and configured to illuminate an upper end portion of the rack, a measuring camera disposed at the tub and capturing an image of the upper end portion of the rack and generating an object image, and a controller configured to perform a first lighting control on the lighting devices, acquiring a first object image generated based on a first image of the rack captured by the measuring camera, performing a second lighting control on the lighting devices, acquiring a second object image generated based on a second image of the upper end portion of the rack captured by the object measuring camera, and generating a three-dimensional shape image of the object based on the first and second object images.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 10-2020-0121679, filed on Sep. 21, 2020, the disclosureof which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to a dishwasher and a method foracquiring a three-dimensional image thereof.

BACKGROUND

A conventional dishwasher can remove leftovers and contaminants oncooking vessels and wash the cooking vessels by spraying high-pressurewash water after meals. The dishwasher perform a variety of steps towash cooking vessels. In the first step, the dishwasher removesleftovers on cooking vessels by using friction between the cookingvessels and wash water without detergent simply by spraying the washwater, and then perform steps of main washing, rinsing, heating andrinsing, drying and the like to wash the cooking vessels completely.However, a degree to which cooking vessels are washed varies dependingon the volume and the types of the cooking vessels placed in thedishwasher, positions at which the cooking vessels are held, and thelike. Hereinafter, a structure of a conventional dishwasher is describedwith reference to FIG. 1 .

FIG. 1 is a diagram illustrating a conventional dishwasher with a dooropen.

Referring to FIG. 1 , an exterior of the dishwasher is comprised of acase 1 with a front open, and a door 2 configured to close the openfront of the case. The case 1 and the door 2 at a front of thedishwasher can cover an inner space of the dishwasher to prevent washwater or detergent from escaping outside of the dishwasher while cookingvessels are washed.

The door 2 can be open while forming an angle of 90 degrees between thedoor 2 and the case 1. The dishwasher is provided with a tub 18 that isconfigured to accommodate wash water, a sump 16 that is disposed at alower side of the tub and that is configured to collect wash water,filter foreign substances from the wash water, and spray the filteredwash water, a nozzle 14 connected to the sump and configured to spraythe wash water into the tub while rotating, and racks 11 and 12 providedin the tub, having upper and lower racks, and configured to store aplurality of cooking vessels.

Leftovers having a large volume are first removed from the cookingvessels and collected on a surface of a lower side of the tub of thedishwasher. Then as the dishwasher proceeds with the following steps,leftovers having a small volume are removed from the cooking vessel oneby one such that the cooking vessels are washed, and the removedleftovers are collected on the surface of the lower side of the tub ofthe dishwasher.

The dishwasher proceeds with the predetermined steps consecutivelyregardless of the types or volume of cooking vessels kept on the rack.For example, the predetermined steps do not include measuring the volumeof the cooking vessels. Accordingly, when the volume of the cookingvessels accommodated is greater than a recommended volume, the cookingvessels are less likely to be washed completely. In this case, to washthe cooking vessels completely, the dishwasher may have to be operatedagain that would result wasting more energy. When the volume of thecooking vessels accommodated is less than the recommended volume, thepredetermined steps may still waste energy even though the cookingvessels are washed sufficiently.

Additionally, a degree to which the cooking vessels are washed variesdepending on a position of the cooking vessels on the rack and how thecooing vessels are held. For example, the cooking vessels held at aslant in an edge portion of the rack are less likely than the cookingvessels laid at a center of the rack with their concave portion facingdown to be washed completely. To prevent this from happening, a user mayhave to consider a position to place cooking vessels and a direction inwhich the cooking vessels are held. Additionally, when the user placesthe cooking vessels in any position of the rack, or when the cookingvessels are moved to a corner of the rack due to hydraulic pressure ofwash water, the cooking vessels are not completely washed.

SUMMARY

The present disclosure is directed to generate three-dimensional shapesof cooking vessels kept on a rack, perform different washing stepsdepending on the types and volume of the cooking vessels kept on therack, and save energy based on an efficient wash.

The present disclosure is also directed to ascertain the positions,density, and the like of the cooking vessels kept on the rack, controlan amount of wash water to be used, wash time, and a wash direction, andimprove wash performance based on intensive wash toward a position wherethe cooking vessels are placed.

According to one aspect of the subject matter described in thisapplication, a dishwasher can include a case that defines an exterior ofthe dishwasher and that has an opening at a front side of the case, adoor configured to open and close the opening of the case, a tub that isconfigured to receive water, a sump that is disposed in the tub, a rackthat is disposed in the sump and that is configured to receive anobject, a plurality of lighting devices that are disposed at a firstposition of an inner surface of the tub and that are configured toilluminate an upper end portion of the rack, a measuring camera that isdisposed at a second position of the inner surface of the tub and thatis configured to capture an image of the upper end portion of the rackand generate an object image, and a controller. The controller can beconfigured to control operations of the dishwasher by: performing afirst lighting control on the plurality of lighting devices toilluminate the upper end portion of the rack, acquiring, based on thefirst lighting control being performed, a first object image generatedas a result of capturing of a first image of the upper end portion ofthe rack by the measuring camera, performing a second lighting controlover the plurality of lighting devices to illuminate the upper endportion of the rack, acquiring, based on the second lighting controlbeing performed, a second object image generated as a result ofcapturing of a second image of the upper end portion of the rack by theobject measuring camera, and generating a three-dimensional shape imageof the object in the rack based on the first object image and the secondobject image.

Implementations according to this aspect can include one or more of thefollowing features. For example, the controller can be configured tocalculate a lighting distance from each pixel included in the firstobject image or the second object image, generate a reference point byusing a coordinate of the measuring camera, coordinates of the pluralityof lighting devices, and the lighting distance, calculate a coordinateof a candidate reflection point in a three-dimensional space by usingthe coordinate of the measuring camera and the reference point,determine a final reflection point based on the candidate reflectionpoint, and generate a three-dimensional shape image of the object basedon the determined final reflection point.

In some examples, the controller can be configured to calculate a firstlighting distance based on a first pixel that is a single pixel selectedfrom pixels included in the first object image, and calculate a secondlighting distance based on a second pixel that is at the same positionas the first pixel among pixels included in the second object image. Insome examples, the controller can be configured to extract a first flatsurface including the measuring camera and a first lighting device and asecond lighting device of the plurality of lighting devices, andgenerate a first reference point and a second reference point by usingthe first lighting distance, the second lighting distance, and the firstflat surface.

In some implementations, the controller can be configured to extract asecond flat surface that is defined by a set of points at a centerbetween a three-dimensional coordinate of the first reference point anda three-dimensional coordinate of the measuring camera, and calculate acoordinate of a first candidate reflection point in thethree-dimensional space based on the measuring camera and the secondflat surface. In some examples, the controller can be configured tocalculate a coordinate of a second reference point in thethree-dimensional space by using the first lighting distance, the secondlighting distance, and the first flat surface, extract a third flatsurface that is defined by a set of points at a center between athree-dimensional coordinate of the second reference point and athree-dimensional coordinate of the measuring camera, and calculate acoordinate of a second candidate reflection point in thethree-dimensional space based on the measuring camera and the third flatsurface.

In some examples, the controller can be configured to determine a firstcross point and a second cross point that are defined at cross pointswhere (i) points placed in the first flat surface and at a firstlighting distance from the first lighting device meet (ii) points placedin the first flat surface and at a second lighting distance from thesecond lighting device, a first distance from the first cross point tothe measuring camera being greater than a second distance from thesecond cross point to the measuring camera, determine the first crosspoint as the first reference point, calculate a coordinate of the firstreference point, and determine the second cross point as the secondreference point, and calculate a coordinate of the second referencepoint.

In some examples, the controller can be configured to calculate a firstnormal vector connecting the first candidate reflection point and afirst midpoint, calculate a second normal vector connecting the secondcandidate reflection point and a second midpoint, calculate a thirdnormal vector from a third pixel and calculate a fifth normal vectorfrom a fifth pixel in terms of the third pixel and the fifth pixelconsecutively adjacent to the first pixel, calculate a fourth normalvector from a fourth pixel and calculate a sixth normal vector from asixth pixel in terms of the fourth pixel and the sixth pixelconsecutively adjacent to the second pixel, calculate a first anglevalue based on a comparison between the first normal vector and thethird normal vector, and calculate a second angle value based on acomparison between the second normal vector and the fourth normalvector.

In some implementations, the controller can be configured to calculate athird angle value based on a comparison between the third normal vectorand the fifth normal vector, calculate a fourth angle value based on acomparison between the fourth normal vector and the sixth normal vector,determine, based on the first angle value matching the third angle valueand the second angle value not matching the fourth angle value, thefirst candidate reflection point as a final reflection point, anddetermine, based on the first angle value not matching the third anglevalue and the second angle value matching the fourth angle value, thesecond candidate reflection point as a final reflection point.

In some implementations, based on the second position being a lateralsurface of the tub, the first position can be above the second position,at least two lighting devices can be disposed at one of two spaces setby a reference surface set in the tub, and at least three lightingdevices can be disposed in the tub. In some implementations, based onthe second position being an upper portion of the tub, the firstposition and the second position can be at the same height with respectto a bottom of the tub, and an angle among the plurality of lightingdevices is 90 degrees or greater with respect to the measuring camera.

In some examples, the first lighting control and the second lightingcontrol can include controlling the plurality of lighting devices suchthat the plurality of lighting devices are consecutively turned on andoff or brightness of the plurality of lighting devices is adjusted.

According to another aspect of the subject matter described in thisapplication, a method for acquiring a three-dimensional image by adishwasher can include performing a first lighting control on aplurality of lighting devices disposed at a first position of an innersurface of a tub, acquiring, based on the first lighting control beingperformed, a first object image generated as a result of capturing of afirst image of an upper end portion of a rack by a measuring cameradisposed at a second position, performing a second lighting control onthe plurality of lighting devices, acquiring, based on the secondlighting control being performed, a second object image generated as aresult of capturing of a second image of the upper end portion of therack by the measuring camera, and generating a three-dimensional shapeimage of an object held in the rack based on the first object image andthe second object image.

Implementations according to this aspect can include one or more of thefollowing features. For example, generating a three-dimensional shapeimage of the object can include calculating a lighting distance fromeach pixel included in the first object image or the second objectimage, generating a reference point by using a coordinate of themeasuring camera, coordinates of the plurality of lighting devices, andthe lighting distance, calculating a coordinate of a candidatereflection point in a three-dimensional space by using the coordinate ofthe measuring camera and the reference point, determining a finalreflection point based on the candidate reflection point, and generatinga three-dimensional shape image of the object based on the determinedfinal reflection point.

In some implementations, calculating a lighting distance from each pixelcan include calculating a first lighting distance based on a first pixelthat is a single pixel selected from pixels included in the first objectimage, and calculating a second lighting distance based on a secondpixel that is at the same position as the first pixel among pixelsincluded in the second object image. In some examples, generating areference point can include extracting a first flat surface includingthe measuring camera and a first lighting device and a second lightingdevice of the plurality of lighting devices, and generating a firstreference point and a second reference point by using the first lightingdistance, the second lighting distance, and the first flat surface.

In some examples, calculating a coordinate of a reflection point in athree-dimensional space can include extracting a second flat surfacethat is defined by a set of points at a center between athree-dimensional coordinate of the first reference point and athree-dimensional coordinate of the measuring camera, and calculating acoordinate of a first candidate reflection point in thethree-dimensional space based on the measuring camera and the secondflat surface.

In some implementations, the method can further include calculating acoordinate of a second reference point in the three-dimensional space byusing the first lighting distance, the second lighting distance, and thefirst flat surface, extracting a third flat surface that is defined by aset of points at a center between a three-dimensional coordinate of thesecond reference point and a three-dimensional coordinate of themeasuring camera, and calculating a coordinate of a second candidatereflection point in the three-dimensional space based on the measuringcamera and the third flat surface. In some examples, calculating acoordinate of a first reference point in the three-dimensional space caninclude determining a first cross point and a second cross point thatare defined at cross points where (i) points placed in the first flatsurface and at a first lighting distance from the first lighting devicemeet (ii) points placed in the first flat surface and at a secondlighting distance from the second lighting device, a first distance fromthe first cross point to the measuring camera being greater than asecond distance from the second cross point to the measuring camera,determining the first cross point as the first reference point,calculating a coordinate of the first reference point, determining thesecond cross point as the second reference point, and calculating acoordinate of the second reference point.

In some implementations, the method can further include calculating afirst normal vector connecting the first candidate reflection point anda first midpoint, calculating a second normal vector connecting thesecond candidate reflection point and a second midpoint, calculating athird normal vector from a third pixel and calculating a fifth normalvector from a fifth pixel in terms of the third pixel and the fifthpixel consecutively adjacent to the first pixel, calculating a fourthnormal vector from a fourth pixel and calculating a sixth normal vectorfrom a sixth pixel in terms of the fourth pixel and the sixth pixelconsecutively adjacent to the second pixel, calculating a first anglevalue based on a comparison between the first normal vector and thethird normal vector, and calculating a second angle value based on acomparison between the second normal vector and the fourth normalvector. In some examples, determining a final reflection point caninclude calculating a third angle value based on a comparison betweenthe third normal vector and the fifth normal vector, calculating afourth angle value based on a comparison between the fourth normalvector and the sixth normal vector, determining, based on the firstangle value matching the third angle value and the second angle valuenot matching the fourth angle value, the first candidate reflectionpoint as a final reflection point, and determining, based on the firstangle value not matching the third angle value and the second anglevalue matching the fourth angle value, the second candidate reflectionpoint as a final reflection point.

In some implementations, based on the second position being a lateralsurface of the tub, the first position can be above the second position,at least two lighting devices can be disposed at one of two spaces setby a reference surface set in the tub, and at least three lightingdevices can be disposed in the tub. In some implementations, based onthe second position being an upper portion of the tub, the firstposition and the second position can be at the same height with respectto a bottom of the tub, and an angle among the plurality of lightingdevices can be 90 degrees or greater with respect to the measuringcamera.

In some examples, the first lighting control and the second lightingcontrol can include controlling the plurality of lighting devices suchthat the plurality of lighting devices are consecutively turned on andoff or brightness of the lighting devices is adjusted.

BRIEF DESCRIPTION OF DRAWING

FIG. 1 is a diagram illustrating a conventional dishwasher with a dooropen.

FIG. 2 is a diagram illustrating a plurality of lighting devices atdifferent positions and a vessel measuring camera capturing an image ofreflected light rays reflecting off a cooking vessel by the plurality oflighting devices.

FIG. 3 is a diagram illustrating amounts of light rays on a surface ofan object when the light rays strike the object.

FIG. 4 is a block diagram illustrating an exemplary dishwasher.

FIG. 5 is a flow chart illustrating a method by which a lightingcontroller turns on and off lighting devices.

FIG. 6 is a flow chart showing a method by which the lighting controllercontrols illuminance of lighting devices.

FIG. 7 is a graph illustrating a distribution of amount of lightaccording to a lighting distance.

FIG. 8 is a diagram illustrating a view of a method by which a candidatereflection point calculator generates first and second reference points.

FIG. 9 is a diagram illustrating a view of a method by which a candidatereflection point calculator generates a first candidate reflection pointby using a second flat surface and a vessel measuring camera.

FIGS. 10A and 10B are diagrams illustrating a view of a method by whicha candidate reflection point calculator generates a candidate reflectionpoint.

FIG. 11 is a diagram illustrating a view of a method by which a finalreflection point determiner determines a final reflection point from afirst candidate reflection point and a second candidate reflectionpoint.

FIG. 12 is a diagram illustrating a view of a position at which thelighting device and the vessel measuring camera are disposed.

FIG. 13 is a flow chart illustrating a method for generating a shape ofa cooking vessel.

DETAILED DESCRIPTION

FIG. 2 is a diagram illustrating a plurality of lighting devices atdifferent positions and a vessel measuring camera capturing an image ofreflected light rays reflecting off a cooking vessel by the plurality oflighting devices, and FIG. 3 is a diagram illustrating amounts of lightrays on a surface of an object when the light rays strike the object.Hereinafter, description is given with reference to FIGS. 2 and 3 .

When light rays approach toward a boundary surface of two mediums, afirst portion of light energy passes through the boundary surface, asecond portion of the light energy is absorbed into the mediums, and athird portion of the light energy reflects. For example, the reflectedlight rays reflect in a reflect direction opposite to a direction inwhich the light rays do not pass through the boundary surface andstrike. The camera can identify the object by using reflected light raysof light rays emitted by the lighting devices. The reflected light raysmay fall into categories of that directly affect the shading of anobject, and scattered light rays that affect the overall imagebrightness through light scattering.

Referring to FIG. 2 , light rays emitted by each of the three lightingdevices 17 a, 17 b, and 17 c strike on a cooking vessel 7 and thenreflect off the cooking vessel 7, and an amount of the reflected lightrays can be captured by a camera P0. In some implementations, each ofthe three lighting devices 17 a, 17 b, and 17 c can be configured toperform turn-on and turn-off operations and when a first lighting device17 a is turned on, a first measured point 8 may be the brightest, and asecond measured point 9 may be the darkest, among measured pointsmeasured by the camera, as a result of direct reflection. When the firstlighting device 17 a is turned off and a third lighting device 17 b isturned on, the second measured point 9 may be the brightest and thefirst measured point 8 may be the darkest among the measured points.When only the second lighting device 17 c is turned on, a centralportion between the first measured point 8 and the second measured point9 may be measured to be the brightest while the first measured point 8may not be affected by direct reflection because of a lighting angle butmay be affected by scattered light.

Referring to FIG. 3 , light rays emitted from the lighting device maydirectly reflect off a first area 10 a and indirectly reflect off asecond area 10 b, and may not reflect off a third area 10 c. An amountof the reflected light rays can decrease gradually from the first area10 a toward the third area 10 c. Accordingly, an amount of light rays ona surface of an object may decrease.

As described above, the amount of the reflected light rays and theamount of the light rays on the surface of the object may vary dependingon an incident angle of light, i.e., a position of the lighting device.Accordingly, a distance from the surface of the object to a center ofthe lighting device can be conversely calculated based on the amount ofthe light rays on the surface of the object. Additionally, since most ofthe cooking vessels stored in the dishwasher are ceramics having asmooth surface, a coordinate of a measured point in a three-dimensionalspace can be ascertained based on total reflection. A method forcalculating the distance to the center of the lighting device isdescribed below.

FIG. 4 is a block diagram illustrating an exemplary dishwasher.

Referring to the drawing, the lighting device 17 can be disposed at afirst position of an inner surface of a tub and light up an upper endportion of a rack. A plurality of lighting devices 17 can be provided,and a controller 20 can control operations of the lighting device 17.

A vessel measuring camera P0 can be disposed at a second position of theinner surface of the tub and capture an image of the upper end portionof the rack to generate an image of vessels. The vessel measuring cameraP0 can include a device configured to convert an external optical imagesignal to an electric image signal by using an image sensor. Forexample, the device can include a complementarymetal-oxide-semiconductor (CMOS) image sensor (CIS) or a charge-coupleddevice (CCD). The first position at which the lighting device 17 isdisposed and the second position at which the vessel measuring camera POis disposed are described below.

The controller 20 can include a lighting controller 21, a data storage22, a lighting distance calculator 24, a candidate reflection pointcalculator 26, a final reflection point determiner 28, and a 3D shapeimage generator 29.

The lighting controller 21 can control the plurality of lightingdevices. Specifically, the lighting controller 21 can turn on and offthe lighting device 17 or control brightness of the lighting device 17.In some implementations, the lighting controller 21 can acquiredifferent vessel images through the vessel measuring camera P0 while theplurality of lighting devices are controlled.

The lighting distance calculator 24 can calculate a lighting distancebased on the vessel image acquired from the vessel measuring camera P0.For example, the lighting distance calculator 24 can calculate an amountof light rays on a surface of a cooking vessel from each pixel includedin the vessel image to calculate a lighting distance of each pixel.

The candidate reflection point calculator 26 can generate a referencepoint by using a predetermined coordinate of the vessel measuring cameraP0, coordinates of the plurality of lighting devices, and lightingdistances in a three-dimensional space, and calculate a coordinate of acandidate reflection point by using the coordinate of the vesselmeasuring camera and the reference point. Additionally, the candidatereflection point calculator 26 can calculate coordinates of one or tworeflection points from one of the pixels included in the vessel image.For example, the candidate reflection point calculator 26 can generatetwo reference points and calculate coordinates of two candidatereflection points when two lighting devices 17 are used, and cangenerate a reference point and calculate a coordinate of a candidatereflection point when three or more lighting devices 17 are used.

When the number of coordinates of a candidate reflection point generatedfrom a single pixel is one, the final reflection point determiner 28 candetermine the generated candidate reflection point as a coordinate of afinal reflection point. When the number of coordinates of a candidatereflection point generated from a single pixel is two, the finalreflection point determiner 28 can determine a coordinate of one of thetwo candidate reflection points as a coordinate of a final reflectionpoint. For example, the final reflection point determiner 28 cancalculate each normal vector by using the candidate reflection pointsgenerated from each pixel, calculate an angle value from each normalvector based on a curve of a surface of a cooking vessel, and determinea final reflection point. A method for determining a final reflectionpoint is described below.

The 3D image generator 29 can combine coordinates of final reflectionpoints generated from each pixel of a vessel image and generate athree-dimensional shape image of a cooking vessel.

FIG. 5 is a flow chart illustrating a method by which a lightingcontroller turns on and off lighting in one embodiment, and FIG. 6 is aflow chart illustrating a method by which the lighting controllercontrols illuminance of lighting in one embodiment. Hereinafter, amethod of controlling lighting is described with reference to FIGS. 2, 5and 6 .

The lighting controller 21 can control the lighting devices 17. Forexample, the lighting controller 21 can perform first lighting controlover the plurality of lighting devices disposed on the inner surface ofthe tub, and when acquiring a first vessel image generated as a resultof the vessel measuring camera P0's capturing of an image of the upperend portion of the rack in the state in which the first lighting controlis performed, the lighting controller 21 can perform second lightingcontrol over the plurality of lighting devices. Then the lightingcontroller 21 can acquire a second vessel image as a result of thevessel measuring camera's capturing of an image of the upper end portionof the rack in the state in which the second lighting control isperformed. The first lighting control and the second lighting controlcan be lighting control including the consecutive turn-on and turn-offof the plurality of lighting devices or the adjustment of brightness ofthe plurality of lighting devices.

FIG. 2 shows three lighting devices 17 a, 17 b, and 17 c lighting up acooking vessel. Hereinafter, three lighting devices are described as anexample of the plurality of lighting devices but more or less lightningdevice can be used. Referring to FIG. 5 , when the first lighting device17 a is powered on in a state in which the second lighting device 17 band the third lighting device 17 c are off (S100), the lightingcontroller 21 can acquire a first vessel image generated as a result ofthe vessel measuring camera P0s' capturing of an image of the upper endportion of the rack (S101), and turn off the first lighting device 17 a(S102). When the second lighting device 17 b is turned on in aconsecutive order (S103), the lighting controller 21 can acquire asecond vessel image (S104) and turn off the second lighting device 17 b(S105). Likewise, when the third lighting device 17 c is turned on(S106), the lighting controller 21 can acquire a third vessel image(S107) and turn off the third lighting device 17 c (S108). The lightingcontroller 21, as described above, can control the lighting devicesdisposed at different angles consecutively. Thus, the controller 20 cangenerate a three-dimensional shape image of a cooking vessel based onthe acquired vessel image and generate a three-dimensional shape imagemore accurately as a result of the acquisition of a plurality of vesselimages.

The lighting controller 21 can control the turn-on and turn-off of thelighting device based on whether power is supplied to the lightingdevice, and brightness of the lighting device based on a power supply inthe state in which power is supplied to the lighting device. Referringto FIG. 6 , the lighting controller 21 can adjust brightness of thefirst lighting device 17 a to a higher brightness level under the firstlighting control (S110) and acquire a first vessel image (S111), andthen adjust the brightness of the first lighting device 17 a to a lowerbrightness level (S112) and adjust brightness of the second lightingdevice 17 b to a higher brightness level under the second lightingcontrol and acquire a second vessel image (S113). Then, the lightingcontroller 21 can acquire a third vessel image (S115) under thirdlighting control (S114) in the same way. Herein, after acquiring thefirst vessel image, the lighting controller 21 can adjust the brightnessof the second lighting device 17 b only to a higher brightness level andacquire a second vessel image in the state in which the brightness ofthe first lighting device 17 a is not adjusted to a lower brightnesslevel. For example, the second vessel image can be acquired in the statein which the brightness of both the first lighting device 17 a and thesecond lighting device 17 b is adjusted to a higher brightness level.

In some implementations, the dishwasher can acquire a three-dimensionalimage based on the acquired vessel images as long as the lightingdevices have different illuminance values as a result of the lightingcontrol regardless of the turn-on and turn-off of the lighting devices.

In some implementations, the first lighting control and the secondlighting control can include controlling a light source frequency of alighting device. For example, the plurality of lighting devices disposedat different positions may not control the brightness of lightingdevices, but may supply power to all of the lighting devices and controleach lighting source frequency, i.e., wavelength of light, to bedifferent. Thus, different vessel images can be acquired depending onlight source frequencies without additional illuminance adjustments.

FIG. 7 is a graph illustrating a distribution of amount of lightaccording to a lighting distance.

In FIG. 7 , a point where the horizontal axis and the vertical axis meetdenotes a center of a light source, i.e., a lighting center 30. Thehorizontal axis denotes a lighting distance (d), which is the distancefrom a lighting device, and the vertical axis denotes an amount of light(W/m2). As shown in FIG. 7 , the amount of light varies according to thedistance from the lighting device, the light amount distribution patternis symmetrical on the left and right with respect to the center oflighting device, and the amount of light decreases as the distance fromthe center of lighting device increases. For example, the center 33 ofthe lighting device where the lighting distance d is 0 has the greatestamount of light, and amount of light is 0 or very small at a point 34that is farthest from the lighting device.

The amount of light can be extracted from each pixel included in avessel image acquired from the vessel measuring camera as a result ofconversion of an optical image signal into an electric image signal.Each pixel has a different distance from a single light source.Accordingly, each pixel may have different amount of light.Additionally, pixels at the same position may have different amount oflight since a different lighting device is used for each vessel image.

Using the relationship between the lighting distance and the amount oflight as shown in FIG. 7 , the lighting distance of the object can becalculated by measuring the amount of light of the object included inthe image.

For example, assume that first and second vessel images are acquired asa result of control over two lighting devices. The lighting distancecalculator 24 can calculate a first lighting distance d1 correspondingto the amount of light 31 of a first pixel that is any pixel included ina first vessel image based on distribution of amount of light asillustrated in FIG. 7 , and calculate a second lighting distance d2corresponding to the amount of light 32 of a second pixel that is apixel at the same position as the first pixel among pixels included inthe second vessel image. Then the candidate reflection point calculator26 can generate a reference point prior to generation of a candidatereflection point.

FIG. 8 is a diagram illustrating a view of a method by which a candidatereflection point calculator generates first and second reference points,and FIG. 9 is a diagram illustrating a view of a method by which acandidate reflection point calculator generates a first candidatereflection point by using a second flat surface and a vessel measuringcamera. Hereinafter, description is given with reference to FIGS. 8 and9 .

Referring to the drawing, in a three-dimensional space, a coordinate ofthe camera may be P0 (x0, y0, z0), a coordinate of the first lightingdevice may be P1 (x1, y1, z1), a coordinate of the second lightingdevice may be P2 (x2, y2, z2), and a coordinate of the third lightingdevice may be P3 (x3, y3, z3). The coordinates P0, P1, and P2 arerespectively a predetermined value when the camera and the lightingdevices are disposed for the first time. The candidate reflection pointcalculator 26 can generate a first flat surface Al that is a singlevirtual flat surface including P0, P1, and P2.

In this case, in the three-dimensional space, when a set of pointslocated at a first center distance d1 from P1 and a set of pointslocated at a second center distance d2 from P2 meet each other to form acircle, the points located on the first plane A1 may only be P3 (x3, y3,z3) and P4 (x4, y4, z4).

Of the two cross points where points placed in the first flat surfaceand placed at the first lighting distance from the first lighting devicemeet points placed in the first flat surface and placed at the secondlighting distance from the second lighting device, a cross point farfrom the vessel measuring camera is determined as a first referencepoint P3, and a three-dimensional coordinate P3 (x3, y3, z3) of thefirst reference point can be calculated. Additionally, of the two crosspoints where points placed in the first flat surface and placed at thefirst lighting distance from the first lighting device meet pointsplaced in the first flat surface and placed at the second lightingdistance from the second lighting device, a cross point near the vesselmeasuring camera can be determined as a second reference point P4, and athree-dimensional coordinate P4 (x4, y4, z4) of the second referencepoint P4 can be calculated.

In some implementations, coordinate values of two reference points arecalculated in a three-dimensional space using two lighting devices. Insome implementations, the coordinate value of the reference point can becalculated using three or more lighting devices. In this case, only thecoordinate value of one reference point may be calculated by using thecenter value of the plurality of cross points as the coordinate value ofthe reference point.

FIG. 9 shows a reflection point Pr1 (xr1, yr1, zr1), a position P0 (x0,y0, z0) of the camera, a first reference point P3 (x3, y3, z3), a secondflat surface A2, and a cooking vessel 80 laid with its convex surfacefacing up.

The second flat surface A2 is a virtual flat surface and denotes a setof points at the same distance between the camera P0 and the firstreference point P3. Of points on the second flat surface A2, a point ata center of a linear distance from the vessel measuring camera P0 to thefirst reference point P3 is a first midpoint Pn1.

The candidate reflection point calculator 26 can calculate a coordinatePr1 (xr1, yr1, zr1) of a first candidate reflection point by using thevessel measuring camera P0 and the second flat surface A2.

The candidate reflection point calculator 26 can extract a secondmidpoint P4 at a center of a linear distance from the camera P0 to thesecond reference point P4 and a third flat surface A3 that is a set ofpoints at the same distance from the second reference point P4 and thevessel measuring camera P0. Additionally, of points on the third flatsurface A3, a point at the center of the linear distance from the vesselmeasuring camera P0 to the second reference point P4 is a secondmidpoint Pn2. The candidate reflection point calculator 26 can calculatea coordinate Pr2 (xr2, yr2, zr2) of a second candidate reflection pointby using the vessel measuring camera P0 and the third flat surface A3.

FIGS. 10A and 10B are diagrams illustrating a view of a method by whicha candidate reflection point calculator generates a candidate reflectionpoint.

From a geometric perspective, positions at which points in athree-dimensional space are formed on a two-dimensional image can bedetermined based on a position and a direction of a camera at the timewhen the camera captures the image. The image captured by the camera isacquired as a result of projection of the points in thethree-dimensional space onto the flat surface of the two-dimensionalimage. Conversely, a coordinate of the three-dimensional space can beacquired from the two-dimensional image as a result of cameracalibration. In some implementations, the camera calibration process andthe pinhole camera projection model may be applied to calculate acoordinate of a candidate reflection point in the three-dimensionalspace from a vessel image.

FIG. 10A is a diagram illustrating a perspective view of a relationshipbetween a position P0 of the vessel measuring camera and the second flatsurface A2, and FIG. 10B is a diagram illustrating a side view of arelationship between a position P0 of the vessel measuring camera andthe second flat surface A2.

In FIGS. 10A and 10B, an optic axis direction in front of the vesselmeasuring camera is set to a Z-axis, an up-down direction of the camerais set to a Y-axis, and a left-right direction of the camera is set toan X-axis, with respect to a focal point P0 of the vessel measuringcamera as an origin. FIG. 10A depicts a coordinate Pr1 (xr1, yr1, zr1)in a three-dimensional space and, an origin Pn1 (0,0) (i.e., a secondmidpoint) on the second flat surface A2, and a coordinate Pr1 (x,y) ofthe second flat surface, and f denotes a distance from a center of alens of the vessel measuring camera to the second flat surface A2 (seeFIG. 10B). In this case, a relationship between Pr1(x, y) of the secondflat surface on which the image is projected in response to thecoordinates Pr1 (xr1, yr1, zr1) in the three-dimensional space may beexpressed as equation 1 hereunder.f:zr1=x:xr1, f:zr1=y:yr1   [Equation 1]

Using equation 1, the coordinate Pr1 (xr1, yr1, zr1) of the firstcandidate reflection point in the three-dimensional space can becalculated based on the coordinate Pr1(x,y) on the second flat surfaceA2.

Likewise, using equation 2, the coordinate Pr2 (xr2, yr2, zr2) of thesecond candidate reflection point in the three-dimensional space can becalculated based on a coordinate Pr2 (x,y) of the third flat surface A3.f:zr2=x:xr2, f:zr2=y:yr2   [Equation 2]

The coordinate of the first candidate reflection point and thecoordinate of the second candidate reflection point, calculated by usingthe above equations, may differ from each other. Accordingly, one ofcoordinate of the first candidate reflection point and the coordinate ofthe second candidate reflection point may be determined as a coordinateof a final reflection point. If two or more lighting devices are usedand a single reference point is only generated, a single candidatereflection point is only generated. Accordingly, the generated candidatereflection point can be determined as a final reflection point.

FIG. 11 is a diagram illustrating a view of a method by which a finalreflection point determiner determines a final reflection point from afirst candidate reflection point and a second candidate reflectionpoint.

A three-dimensional shape image of a cooking vessel can be generated bycombining candidate reflection points of all pixels. In this case, asurface of the same cooking vessel may look convex or concave due toperspective, causing a distortion. To remove the distortion, the finalreflection point determiner 28 can determine a final reflection pointbased on continuity of each successive pixel of the first candidatereflection points and the second candidate reflection points. Forexample, continuity denotes continuity of a position and continuity ofan angle, and to remove a distortion, continuity of an angle is needed.To determine continuity, a normal vector needs to be first calculatedfrom each candidate reflection point.

Referring to FIGS. 9 and 11 , the final reflection point determiner 28can calculate a first normal vector V1 directed from the first midpointPn1 (xn1, yn1, zn1) to the first candidate reflection point Pr1 (xr1,yr1, zr1) in a perpendicular direction and calculate a second normalvector V2 directed from the second midpoint Pn2 (xn2, yn2, m2) to thesecond candidate reflection point Pr2 (xr2, yr2, zr2) in a perpendiculardirection. For example, the first normal vector and the second normalvector can be calculated from a first pixel that is any pixel amongpixels included in a vessel image.

In some implementations, a third normal vector V3 and a fourth normalvector V4 can be calculated from a second pixel adjacent to the firstpixel, and a fifth normal vector V5 and a sixth normal vector V6 can becalculated from a third pixel adjacent to the second pixel and oppositethe first pixel.

A vector refers to a physical quantity and has a size and direction.Accordingly, as a result of comparison between the first normal vectorV1 and the third normal vector V3 adjacent to the first normal vector,calculated as described above, a first angle value can be calculated,and as a result of comparison between the second normal vector V2 andthe fourth normal vector V4 adjacent to the second normal vector, asecond angle value can be calculated.

Then, as a result of comparison between the third normal vector V3 andthe fifth normal vector V5 adjacent to the third normal vector, a thirdangle value can be calculated, and the calculated third angle value canbe compared with the first angle value. Additionally, as a result ofcomparison between the fourth normal vector V2 and the sixth normalvector V6 adjacent to the fourth normal vector, a fourth angle value canbe calculated, and the calculated fourth angle value can be comparedwith the second angle value.

At a candidate reflection point where a distortion occurs, an anglevalue may not be maintained. Accordingly, a candidate reflection pointwhere an angle value remains constant can be determined as a finalreflection point. For example, when the first angle value matches thethird angle value and the second angle value does not match the fourthangle value, the first candidate reflection point is determined as afinal reflection point, and when the first angle value does not matchthe third angle value and the second angle value matches the fourthangle value, the second candidate reflection point is determined as afinal reflection point.

When the candidate reflection point belongs to the boundary point of thecooking vessel, the first angle value does not match the third anglevalue and the second angle value does not match the fourth angle value.In this case, the final reflection point determiner 28 can search edgeof the cooking vessel and limit a region of interest (ROI) to correctthe error. As described above, a distortion can be removed bycalculating normal vectors and angle values from candidate reflectionpoints. Thus, the 3D image generator 29 can generate an accuratethree-dimensional shape image of a cooking vessel.

FIG. 12 is a diagram illustrating a view of a position at which thelighting device and the vessel measuring camera are disposed.

In some implementations, the tub in the dishwasher is a sealed space.When a lighting device lights up only a portion of a cooking vessel ordisposed at a lower position than the vessel measuring camera, a blindspot where light rays do not reflect may be created, and athree-dimensional shape image of the cooking vessel may not begenerated. Thus, constraints may be imposed on the vessel measuringcamera P0 and the lighting devices 17.

For example, when a single lighting device is provided, or when thelighting devices gather near the vessel measuring camera, or when thevessel measuring camera is disposed at a position much lower than aposition of the lighting device, a three-dimensional shape image of thecooking vessel may not be generated.

Referring to the drawing, a first arrangement 90 a, a second arrangement90 b, and a third arrangement 90 c respectively indicate optimalpositions of the lighting device and the vessel measuring camera. Aplurality of lighting devices 17 is provided and disposed at a firstposition of the inner surface of the tub in the form of a spot, a belt,or a band. A single vessel measuring camera P0 is provided and disposedat a second position of the inner surface of the tub.

The second position of the vessel measuring camera may be an upperportion or a lateral surface of the tub. The first arrangement 90 a andthe second arrangement 90 b are shown when the second position is theupper portion of the tub, and the third arrangement 90 c is shown whenthe second position is the lateral surface of the tub.

In the first arrangement 90 a, the second position and the firstposition are at the same height with respect to a bottom of the tub, andin the second arrangement 90 b, the first position is lower than thesecond position with respect to the bottom of the tub. In this case,there may be constraints. For example, two or more lighting devices needto be provided as the plurality of lighting devices, and an angle amongthe plurality of lighting devices need to be 90 degrees or greater withrespect to the vessel measuring camera. By way of further example,another lighting device needs to be placed on the opposite side of thelighting device with respect to the vessel measuring camera.

When the vessel measuring camera P0 and the lighting devices 17 aredisposed as in the first arrangement 90 a, an error that occurs whengenerating a three-dimensional image of a cooking vessel can bedecreased, since the lighting devices 17 light up the entire upper endportion of the rack and all the lighting devices are on the same flatsurface. Additionally, when the first position is lower than the secondposition as in the second arrangement 90 b, an error may be avoidedfurther since an angle between the vessel measuring camera P0 and thelighting device 17 increases.

In the third arrangement 90 c, there may be constraints. For example,the first position is higher than the second position, at least twolighting devices 17 are disposed in one of the two spaces set by areference surface set in the tub, and at least three lighting devicesmay be disposed in the tub.

When the vessel measuring camera P0 and the lighting device 17 aredisposed as in the third arrangement 90 c, a far distance lightingdevice and a near distance lighting device may all be used. Thus, anaccurate three-dimensional shape image of a cooking vessel can begenerated as a result of a variety of responses regardless of a positionand an arrangement of the cooking vessel.

FIG. 13 is a flow chart illustrating a method for generating a shape ofa cooking vessel.

A controller 20 can calculate a first center distance from a first pixelthat is any pixel among pixels included in a first vessel image (S200),and a second center distance from a pixel at the same position as thefirst pixel among pixels included in a second vessel image (S210). Thenthe controller 20 can generate a first flat surface A1 including avessel measuring camera P0, a first lighting device 17 a, and a secondlighting device 17 b (S220).

The controller 20 can generate a first reference point P3 and a secondreference point P4 on the first flat surface Al from a set of points atthe first center distance from the first lighting device 17 a and a setof points at the second center distance from the second lighting device17 b in a three-dimensional space (S230).

The controller 20 can generate a second flat surface A2 that is a set ofpoints at the same distance between the first reference point P3 and thevessel measuring camera P0, and generate a third flat surface A3 that isa set of points at the same distance between the second reference pointP4 and the vessel measuring camera P0 (S240).

The controller 20 can generate a first candidate reflection point Pr1based on the vessel measuring camera P0 and the second flat surface A2,and a second candidate reflection point Pr1 based on the vesselmeasuring camera P0 and the third flat surface A3 (S250).

The controller 20 may generate a first midpoint Pn1 at a center of alinear distance between the first reference point P3 and the vesselmeasuring camera P0, and a second midpoint Pn2 at a center of a lineardistance between the second reference point P4 and the vessel measuringcamera P0 (S260).

The controller 20 can calculate a first normal vector V1 in a directionfrom the first candidate reflection point Pr1 of the first pixel to thefirst midpoint Pn1, and a second normal vector V1 in a direction fromthe second candidate reflection point Pr2 of the first pixel to thesecond midpoint Pn2 (S270). Then the controller 20 can calculate a thirdnormal vector V3 and a fourth normal vector V4 from a second pixeladjacent to the first pixel, and a fifth normal vector V5 and a sixthnormal vector V6 from a third pixel adjacent to the second pixel andopposite the first pixel, as described above (S280).

The controller 20 can calculate a first angle value as a result ofcomparison between the first normal vector V1 and the third normalvector V3, and when a second angle value calculated as a result ofcomparison between the third normal vector V3 and the fifth normalvector V5 matches the first angle value, the controller 20 can determinethe first candidate reflection point Pr1 as a final reflection point. Insome implementations, the controller 20 can calculate a third anglevalue as a result of comparison between the second normal vector V2 andthe fourth normal vector V4, and when a fourth angle value calculated asa result of comparison between the fourth normal vector V4 and the sixthnormal vector V6 matches the third angle value, the controller 20 candetermine the second candidate reflection point Pr2 as a finalreflection point Pr (S290). Since continuity of adjacent pixels isdetermined by using normal vectors, a distortion can be removed, and anaccurate three-dimensional image of a cooking vessel can be generated.The controller 20 can generate a three-dimensional image of a cookingvessel by combining the final reflection points Pr.

What is claimed is:
 1. A dishwasher, comprising: a case that defines anexterior of the dishwasher and that has an opening at a front side ofthe case; a door configured to open and close the opening of the case; atub that is configured to receive water; a sump that is disposed in thetub; a rack that is disposed in the sump and that is configured toreceive an object; a plurality of lighting devices that are disposed ata first position of an inner surface of the tub and that are configuredto illuminate the rack; a measuring camera that is disposed at a secondposition of the inner surface of the tub and that is configured tocapture an image of the upper end portion of the rack and generate anobject image; and a controller configured to control operations of thedishwasher by: performing a first lighting control on a first subset oflighting devices among the plurality of lighting devices to illuminatethe rack, acquiring, based on the first lighting control beingperformed, a first object image generated as a result of capturing of afirst image of the rack by the measuring camera, performing a secondlighting control on a second subset of lighting devices among theplurality of lighting devices to illuminate the rack, acquiring, basedon the second lighting control being performed, a second object imagegenerated as a result of capturing of a second image of the rack by themeasuring camera, and generating a three-dimensional shape image of theobject in the rack based on the first object image and the second objectimage, wherein the measuring camera is configured to capture the firstobject image and the second object image, and wherein the first subsetof lighting devices and the second subset of lighting devices aredifferent from each other.
 2. The dishwasher of claim 1, wherein thecontroller is configured to: calculate a lighting distance from eachpixel included in the first object image or the second object image,generate a reference point by using a coordinate of the measuringcamera, coordinates of the plurality of lighting devices, and thelighting distance, calculate a coordinate of a candidate reflectionpoint in a three-dimensional space by using the coordinate of themeasuring camera and the reference point, determine a final reflectionpoint based on the candidate reflection point, and generate thethree-dimensional shape image of the object based on the determinedfinal reflection point.
 3. The dishwasher of claim 2, wherein thecontroller is configured to: calculate a first lighting distance basedon a first pixel that is a single pixel selected from pixels included inthe first object image, calculate a second lighting distance based on asecond pixel that is at the same position as the first pixel amongpixels included in the second object image, extract a first flat surfaceincluding the measuring camera and a first lighting device and a secondlighting device of the plurality of lighting devices, and generate afirst reference point and a second reference point by using the firstlighting distance, the second lighting distance, and the first flatsurface.
 4. The dishwasher of claim 3, wherein the controller isconfigured to: extract a second flat surface that is defined by a set ofpoints at a center between a three-dimensional coordinate of the firstreference point and a three-dimensional coordinate of the measuringcamera, calculate a coordinate of a first candidate reflection point inthe three-dimensional space based on the measuring camera and the secondflat surface, calculate a coordinate of a second reference point in thethree-dimensional space by using the first lighting distance, the secondlighting distance, and the first flat surface, extract a third flatsurface that is defined by a set of points at a center between athree-dimensional coordinate of the second reference point and athree-dimensional coordinate of the measuring camera, and calculate acoordinate of a second candidate reflection point in thethree-dimensional space based on the measuring camera and the third flatsurface.
 5. The dishwasher of claim 4, wherein the controller isconfigured to: determine a first cross point and a second cross pointthat are defined at cross points where (i) points placed in the firstflat surface and at a first lighting distance from the first lightingdevice meet (ii) points placed in the first flat surface and at a secondlighting distance from the second lighting device, a first distance fromthe first cross point to the measuring camera being greater than asecond distance from the second cross point to the measuring camera,determine the first cross point as the first reference point, calculatea coordinate of the first reference point, and determine the secondcross point as the second reference point, and calculate a coordinate ofthe second reference point.
 6. The dishwasher of claim 4, wherein thecontroller is configured to: calculate a first normal vector connectingthe first candidate reflection point and a first midpoint, calculate asecond normal vector connecting the second candidate reflection pointand a second midpoint, calculate a third normal vector from a thirdpixel and calculate a fifth normal vector from a fifth pixel in terms ofthe third pixel and the fifth pixel consecutively adjacent to the firstpixel, calculate a fourth normal vector from a fourth pixel andcalculate a sixth normal vector from a sixth pixel in terms of thefourth pixel and the sixth pixel consecutively adjacent to the secondpixel, calculate a first angle value based on a comparison between thefirst normal vector and the third normal vector, calculate a secondangle value based on a comparison between the second normal vector andthe fourth normal vector, calculate a third angle value based on acomparison between the third normal vector and the fifth normal vector,calculate a fourth angle value based on a comparison between the fourthnormal vector and the sixth normal vector, determine, based on the firstangle value matching the third angle value and the second angle valuenot matching the fourth angle value, the first candidate reflectionpoint as a final reflection point, and determine, based on the firstangle value not matching the third angle value and the second anglevalue matching the fourth angle value, the second candidate reflectionpoint as a final reflection point.
 7. The dishwasher of claim 1,wherein, based on the second position being a lateral surface of thetub, the first position is above the second position, at least twolighting devices are disposed at one of two spaces set by a referencesurface set in the tub, and at least three lighting devices are disposedin the tub, and wherein, based on the second position being an upperportion of the tub, the first position and the second position are atthe same height with respect to a bottom of the tub, and an angle amongthe plurality of lighting devices is 90 degrees or greater with respectto the measuring camera.
 8. The dishwasher of claim 1, wherein the firstlighting control and the second lighting control comprise controllingthe plurality of lighting devices such that the plurality of lightingdevices are consecutively turned on and off or brightness of theplurality of lighting devices is adjusted.
 9. A method for acquiring athree-dimensional image by a dishwasher, comprising: performing a firstlighting control on a first subset of lighting devices among a pluralityof lighting devices disposed at a first position of an inner surface ofa tub; acquiring, based on the first lighting control being performed, afirst object image generated as a result of capturing of a first imageof a rack by a measuring camera disposed at a second position;performing a second lighting control on a second subset of lightingdevices among the plurality of lighting devices; acquiring, based on thesecond lighting control being performed, a second object image generatedas a result of capturing of a second image of the rack by the measuringcamera; and generating a three-dimensional shape image of an object heldin the rack based on the first object image and the second object image,wherein the measuring camera is configured to capture the first objectimage and the second object image, and wherein the first subset oflighting devices and the second subset of lighting devices are differentfrom each other.
 10. The method of claim 9, wherein generating athree-dimensional shape image of the object comprises: calculating alighting distance from each pixel included in the first object image orthe second object image, generating a reference point by using acoordinate of the measuring camera, coordinates of the plurality oflighting devices, and the lighting distance, calculating a coordinate ofa candidate reflection point in a three-dimensional space by using thecoordinate of the measuring camera and the reference point, determininga final reflection point based on the candidate reflection point, andgenerating the three-dimensional shape image of the object based on thedetermined final reflection point.
 11. The method of claim 10, whereincalculating a lighting distance from each pixel comprises: calculating afirst lighting distance based on a first pixel that is a single pixelselected from pixels included in the first object image, calculating asecond lighting distance based on a second pixel that is at the sameposition as the first pixel among pixels included in the second objectimage, extracting a first flat surface including the measuring cameraand a first lighting device and a second lighting device of theplurality of lighting devices, and generating a first reference pointand a second reference point by using the first lighting distance, thesecond lighting distance, and the first flat surface.
 12. The method ofclaim 11, wherein calculating a coordinate of a reflection point in athree-dimensional space comprises: extracting a second flat surface thatis defined by a set of points at a center between a three-dimensionalcoordinate of the first reference point and a three-dimensionalcoordinate of the measuring camera, calculating a coordinate of a firstcandidate reflection point in the three-dimensional space based on themeasuring camera and the second flat surface, calculating a coordinateof a second reference point in the three-dimensional space by using thefirst lighting distance, the second lighting distance, and the firstflat surface; extracting a third flat surface that is defined by a setof points at a center between a three-dimensional coordinate of thesecond reference point and a three-dimensional coordinate of themeasuring camera; and calculating a coordinate of a second candidatereflection point in the three-dimensional space based on the measuringcamera and the third flat surface.
 13. The method of claim 12, whereincalculating a coordinate of a first reference point in thethree-dimensional space comprises: determining a first cross point and asecond cross point that are defined at cross points where (i) pointsplaced in the first flat surface and at a first lighting distance fromthe first lighting device meet (ii) points placed in the first flatsurface and at a second lighting distance from the second lightingdevice, a first distance from the first cross point to the measuringcamera being greater than a second distance from the second cross pointto the measuring camera, determining the first cross point as the firstreference point, calculating a coordinate of the first reference point,determining the second cross point as the second reference point, andcalculating a coordinate of the second reference point.
 14. The methodof claim 12, further comprising: calculating a first normal vectorconnecting the first candidate reflection point and a first midpoint;calculating a second normal vector connecting the second candidatereflection point and a second midpoint; calculating a third normalvector from a third pixel and calculating a fifth normal vector from afifth pixel in terms of the third pixel and the fifth pixelconsecutively adjacent to the first pixel; calculating a fourth normalvector from a fourth pixel and calculating a sixth normal vector from asixth pixel in terms of the fourth pixel and the sixth pixelconsecutively adjacent to the second pixel; calculating a first anglevalue based on a comparison between the first normal vector and thethird normal vector; calculating a second angle value based on acomparison between the second normal vector and the fourth normalvector; calculating a third angle value based on a comparison betweenthe third normal vector and the fifth normal vector; calculating afourth angle value based on a comparison between the fourth normalvector and the sixth normal vector; determining, based on the firstangle value matching the third angle value and the second angle valuenot matching the fourth angle value, the first candidate reflectionpoint as a final reflection point; and determining, based on the firstangle value not matching the third angle value and the second anglevalue matching the fourth angle value, the second candidate reflectionpoint as a final reflection point.
 15. The method of claim 9, wherein,based on the second position being a lateral surface of the tub, thefirst position is above the second position, at least two lightingdevices are disposed at one of two spaces set by a reference surface setin the tub, and at least three lighting devices are disposed in the tub,and wherein, based on the second position being an upper portion of thetub, the first position and the second position are at the same heightwith respect to a bottom of the tub, and an angle among the plurality oflighting devices is 90 degrees or greater with respect to the measuringcamera.
 16. The method of claim 9, wherein the first lighting controland the second lighting control comprise controlling the plurality oflighting devices such that the plurality of lighting devices areconsecutively turned on and off or brightness of the lighting devices isadjusted.
 17. The dishwasher of claim 1, wherein the measuring camera isconfigured to, based on the first lighting control being performed bythe first subset of lighting devices and the second lighting controlbeing performed by the second subset of lighting devices, capture thefirst image and the second image at different times.
 18. The method ofclaim 9, wherein the first image and the second image are captured atdifferent times based on the first lighting control being performed bythe first subset of lighting devices and the second lighting controlbeing performed by the second subset of lighting devices.